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1. Introduction 


This document is the final report for the NASA/Langley contract 
entitled "Perceptual Factors that Influence Use of Computer Enhanced 
Visual Displays," performed for Mr. James Burley. 

The document consists of two parts. The first part contains a 
discussion of the problem to which the grant was addressed, a brief 
discussion of work performed under the grant, and several issues 
suggested for follow-on work. 

The second part, presented as Appendix I, contains the annual report 
produced by Dr. Ann Fulop, the Postdoctoral Research Associate who 
worked on-site in this project. 

The main focus of this project was to investigate perceptual factors 
that might affect a pilot's ability to use computer generated 
information that is projected into the same visual space that contains 
information about real world objects. For example, computer 
generated visual information can identify the type of an attacking 
aircraft, or its likely trajectory. 

Such computer generated information must not be so bright that it 
adversely affects a pilot's ability to perceive other potential threats 
in the same volume of space. Or, perceptual attributes of computer 
generated and real display components should not contradict each 
other in ways that lead to problems of accommodation and, thus, 
distance judgments. The purpose of the research carried out under 
this contract was to begin to explore the perceptual factors that 
contribute to effective use of these displays. 


2. Work Pe rformed Under the Contract 

The initial year of this project focused on two primary topics, which 
are described in detail in the Annual Report, included as Appendix I. 
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First, the post-doctoral research associate (Ann Fulop) and the 
domain expert (Joseph Clark) performed a task analysis of a one-on- 
one air combat engagement. The purposes of the study were to 1) 
explore types of information required by pilots in one-on-one 
combat situations in thrust-vectored aircraft and 2) identify 
potential representations of information that display designers might 
use to construct effective displays. 

Second, a study was carried out to determine the relative effects of 
size, distance, and field position of a virtual target on detection of 
those targets in a dual- task human performance study. 

The results of both of these activities are described in Appendix I. 


3. Recommendatio ns for Future Work 

This section contains recommendations for follow-on work. Two 
categories of follow-on work are recommended. The first category 
continues the work begun during the first year. This work would 
continue to focus on perceptual effects of attributes of computer 
generated displays that are overlaid with real-world information. 

The second category of suggested future work addresses the effects 
of cognitive factors such as training and task compatibility on the 
ability to make use of computer generated information that is 
overlaid with real world information in the same virtual space. A 
subsection which discusses previous, relate work performed by the 
Principal Investigator attempts to provide justification for the 
recommendations. 

3.1. Perceptual Factors 

The annual report prepared by Ann Fulop describes several key 
areas of importance for follow-on work, in particular, the effects of 
availability of various cues on distance and size judgments The 
primary topics identified in the report were the effects of 1) various 
patterns of interposition, 2) field of view, and 3) long term exposure 
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to head mounted displays, especially as this variable affects 
convergence. Two cognitive factors were identified in the 
recommendations, namely the effects of various representations of 
abstract concepts associated with (referred to as conformal displays 
in Appendix I.) 

In addition to the factors suggested in the annual report, there are 
other perceptual factors that may be useful to address in follow-on 
work. First, both rapid and slow changes in ambient light may 
affect all the variables identified by Ann Fulop in Appendix I. 

If real-world information is transmitted as part of the display, rapid 
changes in ambient light -- such as the changes generated by an 
explosion of an aircraft or target, or a burst of reflected sunlight -- 
may affect the pilot's ability make use of computer generated 
information. As well, slow changes due to e. g., daybreak or nightfall 
may also affect the ability of pilots to use displays that combine real 
information with computer generated information. 

One potentially useful outcome of this line of research might be the 
specification of equations that can determine how to automatically 
adjust display intensities to offset illumination changes so that both 
the real information and the computer generated information 
maintain constant apparent (relative) brightness. 

Second, the effects of rapid deceleration and acceleration may also 
affect the ability of pilots to use computer generated information. 
The effects of acceleration and deceleration may create 
disorientations that are harder to overcome with computer generated 
displays. Conversely, there may be computer generated displays 
that could offset the effects on real world information of rapid 
acceleration and deceleration. If it were possible to identify such 
offsetting computer generated information, its presentation could be 
triggered under rapid acceleration and deceleration. 
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3.2. Cognitive Factors 


Several cognitive factors may be of importance to performance with 
displays that integrate computer generated information with real 
world information. These suggested studies would address the 
problem of practiced behavior in multi-task settings which combine 
computer generated and real world information. 

This is a potentially useful area of research because expert, multi- 
task behavior may produce effects that differ substantially from 
novice behavior, especially when both groups of subjects are not 
already trained in an existing paradigm. That is, it may be important 
to contrast the effects of various perceptual and cognitive factors on 
performance of 1) skilled pilots who are trained in the new paradigm 
as well as 2) novice pilots who are not highly practiced in the 
existing paradigm. Because new pilots will be increasingly trained in 
the new paradigm, it seems important to understand the effects of 
practice, task compatibility, and so forth on them as well as on 
already- trained pilots. 

3.2.1. Background & Justification of Cognitive Factors 

Beginning in 1973, David Littman and Robert Becklen performed a 
series of experiments under the direction of Ulric Neisser with the 
intention of determining the factors that influence the ability of 
humans to respond effectively to two different, meaningful events 
that occur in the same virtual space. 

The impetus for this series of studies was the demonstration by Paul 
Kolers in 1971 that it was difficult, if not impossible, to follow two 
separate, meaningful, events in the same virtual space. Koler's 
demonstration was performed with an apparatus that consisted of a 
pilot's helmet with the visor replaced with a half-silvered mirror. 
The half-silvered mirror performed two functions. 

• First, the half-silvered mirror permitted the subject to view the 
world in front of him or her. 
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• Second, the half-silvered mirror projected a reflection of the world 
behind the observer into the same virtual space by means of full- 
silvered mirrors placed at various locations on the side of the helmet. 

The observer thus had available at all times the information needed 
to see what was going on in front at any depth, behind at any depth, 
or both. Koler's initial studies with the modified pilot's helmet 
apparatus produced two main findings. 

First, under any reasonable conditions of lighting, including night 
conditions, subjects were able to follow without difficulty either the 
events occurring in front of them or the events occurring behind 
them with essentially no interference. 

Second, novice subjects were utterly unable to follow both the events 
occurring in front of them and the events occurring behind them 
regardless of how hard they tried, how much they were paid, or any 
reasonable manipulations of relative illuminations of the two events. 

That is, as long as the meanings and contents of the events in front of 
and behind the subjects were different, subjects were easily able to 
follow either event without confusion. But they were not able to 
follow both at the same time. Rather, they were forced to "switch" 
back and forth between events whenever they wanted to or were 
instructed to. This suggested that visual attention during meaningful 
tasks may follow many of the same laws as auditory attention during 
meaningful tasks. 

Littman [1] and Littman & Becklen [2] performed a series of 
experiments to determine some of the basic laws that govern the 
abilities of subjects to 1) follow either one or the other of the two 
events, 2) switch from one event to the other, or 3) "see" two events 
at the same time. The results of this research generally agreed with 
the results of Koler's initial studies and, therefore, with models of 
selective auditory attention. Two effects discovered during this 
research may be relevant for the NASA research project. 
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First, subjects got much better at tracking two target events with 
practice. It appeared that, in the initial stages of practice, subjects 
switched back and forth between events, often getting lost when 
they tried to switch back. 1 and often missing targets that were right 
in front of their noses." With practice, however, subjects seemed to 
abandon their switching strategy and were able to identify target 
events in both events simultaneously. The expertise the subjects 
acquired appeared to consist at least in part of identifying 
predictable patterns of activity across the two events that followed 
natural laws of physics. Thus, subjects seemed to be able to learn 
best when the events did not "appear out of nowhere" and where 
patterns of appearance and disappearance became familiar to the 
subjects. In essence, with extensive practice, the subjects appeared 
able to learn to combine two separate events into a single perceptual 
experience. 

Second, subjects also got better at detecting low-probability or even 
totally unexpected events as they acquired expertise in the primary 
task of tracking the visual events. 2 

These two findings together suggest three points: 

• Studies of perceptual ability in certain simple dual task settings 
may yield results that are not completely generalizable to complex 
tasks that are performed under heavy attention loads when subjects 
have had extensive training.. 

• The ability of subjects to identify, categorize, and respond to 
unexpected meaningful events increases significantly with practice. 


iThe two events subjects were asked to monitor were, in some cases, 
distinguished by low level perceptual features, such as the colors of the objects 
or the speed at which they moved. 

2 Again, these totally unexpected events conformed to natural laws of physics. 
Pilot evidence suggested that when unexpected events did not conform to the 
laws of physics (e.g., they appeared out of nowhere) they were much harder to 
detect. 
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• Events that violate natural laws of physics (e.g., they just pop into 
existence) may take longer to learn to detect than events that come 
into existence under natural laws. 

Taken together, these findings from early multi-event tracking 
studies suggest two points that may be of relevance for the problem 
of designing displays that incorporate real and computer generated 
information. 

First, significant aspects of skilled performance of subjects can be 
accounted for by 1) practice and 2) the cognitive compatibility of the 
elements of the displays vis-a-vis the identification, classification, 
and decision making tasks being performed. Thus, perceptual factors 
may have their most significant impact for unpracticed subjects in 
essentially performing largely meaningless tasks. 

Second, the acquisition of the ability to identify "unexpected events" 
appears to follow a step function. Once a subject attains a certain 
level of skill in a primary task, the secondary task e.g., detection of 
unexpected events, becomes very easy. 


4. Final Comments 

The study of the integration of real world and computer generated 
information is a fascinating topic which has implications for many 
domains. Real time decision support systems for battlefield use; 
decision support systems for architects, which, for example, overlay 
computer generated buildings on real terrain; diagnostic systems for 
doctors that permit the diagnostician or surgeon to "look through" 
various layers of a person's body and so forth all will rely on the 
integration of computer generated and real world information. 

The results of studies being carried out at NASA Langley on the 
integration real world and computer generated displays thus have a 
very real chance to be an important subject of technology transfer 
activities. 
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TASK ANALYSIS 
Purposes 

A task analysis of a one versus one air combat situation in which the pilot is in a 
thrust-vectored aircraft was completed. Purposes of the task analysis include: 

1) Determine new problems that pilots will have when maneuvering a thrust- vectored 

aircraft. 

2) Determine how pilots’ information requirements will change in a thrust-vectored 
aircraft. 

3) Inform the display designer of the information requirements and how information 
relates to each other. 

4) Help display designers create performance measures to evaluate display concepts. 

5) Help the display designer exploit the stereo capability of the display medium, or the 

helmet. 

In addition, the information based on the task analysis can be used to: 

1) Acquire a general understanding of air combat. 

2) Facilitate interviews and discussions with pilots. 

3) Facilitate communication among the display designers. 

4) Design displays. 

5) Evaluate display concepts before empirical testing. 

6) Create experimental tasks to test display concepts. 

Method 

A naval aviator, Joseph Clark, was the subject matter expert (SME) for the analysis. The 
SME is a former F-14 pilot with over 100 hours and 300 carrier landings. The subject matter 
expert has spent 100 hours experience in a thrust- vectored flight simulator. 

To begin the analysis, the phases of air combat were described and the goals of each 
phase defined. Table 1 lists the phases and goals of air combat. 

Once the goals of air combat were determined, the tasks the pilots must accomplish to 
achieve each of the goals were listed. This list includes the information pilot’s currently use to 
perform tasks and accomplish goals. In addition, mistakes pilots make were included in the task 
analysis. The information was then grouped into 6 information groups. The groups included 
information concerning weaponeering, spatial awareness, air and surface threats, friendly 
aircraft, target aircraft, and own aircraft. 

A chart was then created to show the relationships among the pieces of information and 
the various tasks involved in air combat. The chart shows all of the information requirements 
on the y-axis and the phases of flight on the x-axis. Looking across the chart the display 
designer knows which information is needed by that pilot during which phases of flight. 
Furthermore, colored boxes indicate which goals pertain to the information. The second column 



Table 1. 

Phases and goals of air combat. 
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I. Pre-merge phase. 

Goal 1: To choose a plan of action to target the highest threat to the completion of the 
mission assigned. 

Goal 2: To choose a plan of action to prevent threat from interfering with mission. 

Goal 3: To choose a plan of action to prevent threat from accomplishing his mission. 

II. Merge Phase. 

Goal 4: To anticipate or predict target’s behavior. 

III. & IV. Initial turn and Second merge. 

Goal 5: To gain an offensive. 

Goal 6: To maintain an offensive 

V. Leave or bug-ouL 

Goal 7: Avoid being shot. 

VI. All phases of mission. 

Goal 8: Enable safe recovery of aircraft. 

Goal 9: Avoid external threat missile envelopes. 

of the chart shows other information that is related to the information in that row. Figure 1. 
shows the chart 

Once the task was completed, 9 pilots reviewed the analysis. The nine pilots included 5 
Navy pilots, 3 Air Force pilots, and 1 NASA test pilot As a group, the pilots have experience in 
all modem fighter aircraft The pilots have spent an average of 10 hours in a thrust-vectored 
simulator. The task analysis was updated to include inputs from the pilots. 

Results 

The task analysis defined the information requirements of a one versus one air combat 
situation and illustrated how information relates to each other. The task analysis also showed 
display designers how pilots visualize and mentally depict concepts. The task analysis created a 
taxonomy of display concepts and information that pilots use during air combat 

The task analysis and chart are available from Joseph Clark, Lockheed Engineering and 
Sciences, Hampton, VA. Any future grantees have this information available for future 
reference. 
























A TARGET DETECTION STUDY FOR DETECTING VIRTUAL 
OBJECTS PRESENTED AT VARYING DEPTHS AND LOCATIONS 
WHILE MONITORING THE PHYSICAL ENVIRONMENT 

ABSTRACT 

The objectives of the present research were to determine whether it was possible for participants 
to monitor simultaneously stimuli in both the virtual world and physical world, and to determine 
the appropriate depth and location to present static virtual objects. Participants were required to 
monitor an arrow projected onto a wall approximately 60 feet in front of them. When they 
detected a change in the direction of the arrow, they responded as quickly as they could by 
pressing a mouse button. At the same time that participants were monitoring the arrow, they 
were looking for a virtual target presented via a helmet-mounted display. Various virtual shapes 
would "pop up" at one of 5 locations and one of 3 depths. When they detected the target 
stimulus, they responded as quickly as they could by pressing a mouse button. Results 
suggested that it was possible for humans to monitor stimuli in both worlds, however reaction 
times were slower than typically reported when monitoring only the physical world. Results 
also showed that virtual objects presented at the same depth as the physical object were detected 
faster than virtual objects presented 3 feet in front of the participants. Virtual stimuli presented 
in the center of the Field of view were responded to faster than stimuli presented at the edges of 
the Field of view. 

INTRODUCTION 

If helmet-mounted displays (HMD) are to have practical applications, it is necessary to 
understand the visual perception problems of intermixing virtual objects presented via the 
binocular HMD with physical objects located in the "real world". HMD’s were developed and 
designed to allow pilots to view symbology at optical inFmity. It was believed that virtual 
imaging devices would enable pilots to simultaneously view displayed symbology and targets 
without needing to change accommodation. However, traditional two-dimensional and biocular 
virtual image displays do not cause the eye to focus at optical inFmity (Hull, Gill, & Roscoe, 

1982; Iavecchia, Iavecchia, & Roscoe, 1988; Normal & Erlich, 1986). With the use of these 
displays, focus shifts from the far point inward toward the dark focus, or the resting 
accommodation. Accommodation depends on the acuity demand of a task. A demanding task 
will pull the eye farther toward the far point than will a less demanding task. Currently, it is 
unknown whether a shift toward the resting accommodation occurs with binocular or stereo 
three-dimensional virtual image displays. The stereo display creates a three-dimensional image 
by presenting a slightly different view to each eye. The brain fuses the disparate images to create 
a single image perceived in depth. Stereo images make it possible to model the physical 
environment and, thus, present images within the display at real world depths. 

Convergence is necessary to fuse the image in a stereo display. However, convergence 
and accommodation are decoupled when viewing objects through an HMD. It is assumed that 
accommodation is Fixed for all objects regardless of the distance at which objects are presented 
through the helmet. For example, when two objects exist at two depths in the virtual display the 
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observer will see double images of the unattended object because both objects are in focus 
simultaneously. The observer cannot converge both objects simultaneously unless they are in 
close range. Thus, when the observer is required to attend to both physical and virtual objects, it 
is unknown how the eyes accommodate and converge. It seems reasonable to assume that when 
the observer is attending to a physical object the eyes are accommodated to that object’s 
distance. Thus, virtual objects presented at the physical object’s distance will also be 
accommodated to that distance and should be responded to faster. However, accommodation to 
virtual objects is assumed to be fixed near the dark focus (Iavecchia et al, 1988) thus, virtual 
objects presented at the dark focus distance may be responded to more quickly than other 
objects. 

It was hypothesized that virtual stimuli presented at the same distance as the physical 
stimuli would be reacted to faster than other stimuli because participants should be accommodated 
at that distance. It was hypothesized that larger stimuli would be responded to faster than the 
smaller stimuli. 

METHOD 

Subjects. 

A total of 14 participants, 1 1 males and 3 females, between the ages of 23 years and 34 
years volunteered to participate in the study. Participants had normal color vision, acuity, and 
stereoacuity. 

Helmet-mounted display (HMD). 

The HMD used was a variable overlap, binocular system with 30 degree vertical by 40 
degree horizontal field-of-view transmissive oculars. The oculars were turned to diverge 5 
degrees horizontally producing a total field-of-view of 45 degrees. The helmet used two 1 inch 
displays (image sources) with a 1280(h) x 1024(v) picture element resolution. The helmet 
weighed 6.5 lbs. 

Task. 

Participants were required to monitor simultaneously a 4 foot arrow projected onto a far wall 57 
feet directly in front of them and virtual objects presented through the helmet mounted display. 
The arrow pointed to the left or right. When the arrow changed direction, the participants were 
required to press the left mouse button as quickly as they could. The arrow changed direction 3 
times per minute. Participants were told that they received 3 points for every change in direction 
that they detected. Six different virtual missile shapes were presented to participants via the 
helmet al approximately the rate of one shape per second. The missile shapes are.shown in 
Figure 1. When they detected the target missile shape, they were required to press the right 
mouse button as quickly as they could. The target was presented 3 times per minute. 

Participants were told that they received one point for every target that they detected. To 
motivate participants to remain focused on the arrow, they received 3 points for detecting a 
change in the arrow direction, but only 1 point for correctly detecting the target. 


Independent variables and experimental design. 
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The three independent variables; distance, location, and size of stimuli were within 
subjects in nature. Virtual stimuli were presented at three different distances, 3.128 feet, 18 feet, 
and 57 feet from the participant. The distance of 3.128 feet is approximately the dark focus 
distance, and 18 feet is approximately the point at which the convergence angle is 0. Virtual 
stimuli were presented at five locations, upper left, upper right, center, lower left, and lower 
right. Thus, stimuli were presented at each of the 5 locations at each of the 3 depths, or 15 
places. The target stimulus was presented in each place 3 times. Virtual stimuli subtended either 
2.5 degrees of arc or 1 degree of arc. The design was a repeated measures design. 


Procedure. 

Participants were read instructions. Then, the helmet was fitted on the participant’s head 
for physical comfort. A standard display (see Figure 2) was used to ensure that participants had 
identical images in both the right and left eyes, and that they were seeing a stereo image. After 
the display was aligned, participants then performed a 5 minute practice task to familiarize 
themselves with the helmet display and the responses. After the practice trial, participants 
performed two 15 minute experimental sessions. In one session, the size of the virtual stimuli 
were 2.5 degrees of arc, in the other session, the virtual stimuli subtended 1 degree of arc. 
Assignment of participants to size condition was counterbalanced. Reaction time data to the 
target stimuli and arrow changes were collected. The experiment was performed in the dark. 

RESULTS 

Mean reaction time (M = 1.095 sec., sd = 2.67) to the change in the arrow suggests that 
participants were primarily attending to the virtual stimuli rather than the arrow change. To test 
the hypotheses, a repeated measures analysis of variance was performed on the reaction time 
data to the target stimulus. Table 1 shows significant main effects for distance, location, and 
size. Significant interaction effects for distance by location, and size by location were found. 
The main effect of distance supports the hypothesis that virtual stimuli presented at the same 
distance as the physical stimulus would be responded to faster. The virtual stimuli presented at 
57 feet (M = 751 ms, sd = 168 ms) were responded to significantly faster than stimuli presented 
at 3. 128 feet (M = 790 ms, sd = 184 ms ). 

The main effect of location shows that virtual stimuli presented in the center of the 
display were responded to faster than stimuli presented in the periphery. Table 2 displays the 
means and standard deviations for reaction time at the five locations. 

The main effect of size shows that the large stimuli (M = 727 ms, sd = 160 ms) were 
responded to faster than the small stimuli (M = 819 ms, sd = 184 ms). This supports the 
hypothesis that larger stimuli would be responded to faster than smaller stimuli. 
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Table 1. Summary table for repeated measures ANOVA on reaction time to target stimuli. 
* Significant at alpha .05. 


SOURCE 

DF 

F 

eta2 

Distance 

2 

6.03* 

.02 

Location 

4 

7.66* 

.04 

Distance by Location 

8 

2.56* 

.03 

Size 

1 

97.45* 

.12 

Size by Distance 

2 

.77 


Size by Location 

4 

2.64* 

.02 

Size by Distance by Loc. 

8 

.41 


Subject 

13 

15.79* 

.23 


Table 2. Mean reaction time of detecting target stimuli at five locations. 



Upper Left 

Upper Right 

Center 

Lower Left 

Lower Right 

Mean 

788 ms 

766 ms 

723 ms 

794 ms 

781 ms 

Standard 

deviation 

197 ms 

165 ms 

152 ms 

179 ms 

183 ms 


The interaction effect of distance by location reveals that stimuli located in the center of 
the display were responded to faster at distances of 16 feet and 57 feet than stimuli in the 
periphery (see Figure 3). However, stimuli located at the center 3 foot distance were not 
responded to faster than stimuli in the periphery. 

The interaction effect of size by location shows that small stimuli presented in the center 
of the display were responded to as quickly as the large stimuli (see Figure 4). 

DISCUSSION 

It was hypothesized that virtual stimuli presented at the same distance as a physical 
stimulus would be reacted to more quickly than virtual stimuli presented at closerdistances. 
Results from the present study supported this hypothesis. Thus, the display designer should 
model the physical world within the display so that important virtual objects are presented at the 
same distance as the physical world target. Furthermore, these results show that displaying 
virtual objects at the dark focus distance results in slower reaction times when the observer is 
attending to a physical objects at a further distance. The increase in reaction time is probably 
due to the need for the eyes to converge and to see objects presented at the dark focus distance. 

It was hypothesized that larger stimuli would be responded to more quickly than smaller 
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Figure 3. - Interaction Effect of Distance by Location for Reaction Time to Target Stimuli 
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Figure 4. Interaction Effect of Size by Location for Reaction Time to Target Stimuli 
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stimuli. Results from the present study showed that stimuli presented in the center of the display 
are responded to faster regardless of the size of the stimulus. This suggests that smaller 
information such as, alphanumerics should be presented in the center of the screen when the 
observer is attending simultaneously to the physical world and virtual world. 

Response times to both the arrow and the virtual missiles were slow. This finding suggests that 
attending to the physical world and the virtual world simultaneously is a difficult task. Thus, 
pilots may have a difficult time switching attention from the virtual scene to the physical scene 
even though they 

simultaneously see both scenes. The problem of cognitively switching from the virtual world to 
the physical world warrants further research. Some participants reported that they did not 
perceive three distances in the display. However, they reported seeing some objects become 
larger or smaller. The only cues to distance in the display were accommodation, convergence, 
and lateral disparity. With HMD’s, accommodation and convergence are decoupled. Thus, 
distance cues need to be included in virtual displays so that observers can accurately determine 
distance and size changes. Future research will explore the effect that pictorial depth cues have 
on distance perception of virtual stimuli. 

REFERENCES 
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VISION SOFTWARE INTERFACE 

An in-house software package , VISION, was created in order to facilitate display design 
for the helmet and testing of displays in the Differential Maneuvering Simulator (DMS). The 
software package allows designers to connect to the DMS, record and playback data runs of 
pilots in the lab, simulate flight while in the lab, and interactively configure helmet displays 
while in the lab. In addition, the software allows the researcher to monitor from a bird’s-eye 
perspective or from an inside the cockpit perspective a pilot’s behavior during an experiment. 

The grantee designed a Motif interface for the software package. The prototype for the 
interface, a User’s Group Description Document, a Task Description Document, an Object 
Description Document, and a Systems Objectives Document were delivered to the software 
developer, Alan Dare. 

CONSULTATION 

Grantee performed experimental method and data analysis service to the cockpit 
technology branch at NASA Langley. Grantee has critiqued and reviewed experimental design 
and helped to design future experiments. In addition, the grantee conducted colloquia to help 
other display designers acquire knowledge and skills in experimental methodology, quantitative 
analysis, and SAS programming. In addition, the grantee provided critiques of various display 
designs and when appropriate made suggestions according to human engineering principles. 

Any future grantee will need to continue these efforts. 
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INTRODUCTION 

Virtual image displays, or head-up combining glass displays (HUD) and helmet-mounted 
projections of computer generated images (HMD), were developed and designed to allow pilots 
to view symbology at optical infinity. This enables pilots to simultaneously view displayed 
symbology and targets without needing to refocus. However, traditional two-dimensional and 
biocular virtual image displays, such as HUDs and HMDs, do not cause the eye to focus at 
optical infinity (Hull, Gill, & Roscoe, 1982; Iavecchia, Iavecchia, & Roscoe, 1988; Norman & 
Erlich, 1986;Roscoe, 1985) With the use of these displays, focus shifts from the far point inward 
toward the dark focus, or the resting accommodation. Consequently, accommodation micropsia, 
"or a reduction in the apparent size of an object of given retinal angle"(Smith, Meehan, & Day, 
1992, pp. 289), causes objects in the physical world to appear smaller and farther away than they 
actually are physically. The micropsia induced by the virtual image causes poor judgments of 
apparent distance and size. This explains why pilots flying with virtual images make fast 
approaches, round out high, and land long and hard (Roscoe, 1987). Controlled crashes into 
terrain during low level flights have also been attributed to the accommodation micropsia 
induced by HUDs. That is, mountain tops seem farther away and lower to the horizon than they 
actually are, the pilot flies at an altitude too low and crashes into the top of the mountain. 
Furthermore, the effect is more pronounced when an observer needs to identify a target on a 
textured terrain than when the terrain is unimportant (Iavecchia et al, 1988). Therefore, adequate 
distance and size judgments are difficult to make with traditional virtual image displays. 

Currently, it is unknown whether accommodation micropsia occurs with binocular or 
stereo three-dimensional virtual image displays. The stereo display creates a three-dimensional 
image by presenting a slightly different view to each eye. The brain fuses the disparate images 
to create a single image perceived in depth. Stereo images make it possible to model the physical 
environment thus present depth cues within the display that model "real world" depth. 

Presenting strong depth cues in the display may eliminate or attenuate the lapse toward resting 
accommodation because objects can be displayed at the same depth as physical "real world" 
objects. Thus, accommodation may shift outward from the dark focus in order to view the 
virtual object at a further distance. Furthermore, the added pictorial cues in the display provide a 
richer textural content in which to make distance and size judgments. As the virtual world 
begins to resemble the physical world size and distance judgments will become more accurate. 

The Zoom-lens hypothesis attempts to explain the accommodation micropsia problem 
with virtual image displays (Roscoe, 1985). The hypothesis states the function of the eye is 
analogous to the zoom-lens of a camera. That is, changes in lens curvature cause changes in the 
size of the retinal image. Inward accommodation from the dark focus produces large changes in 



refraction that bring near objects into sharp focus at the expense of depth of field. These inward 
shifts in accommodation are accompanied by reductions in apparent size. Outward 
accommodation from the dark focus produces small changes in refraction. Image clarity is 
maintained with increasing depth of field, therefore apparent size of objects increases. Zoom-lens 
hypothesis suggests that an outward shift in accommodation also magnifies the retinal image. 
Consequently, more photoreceptors on the retina are stimulated resulting in finer discriminations 
and an increase in apparent size. This hypothesis is supported by the work of Iavecchia et al 
(1988), they showed with increasing acuity demand of a task, accommodation shifted outward. 
The accommodation of the eye is a result of the pull of the dark focus and the pull of the visual 
task. A visually demanding task results in a shift in accommodation away from the dark focus. 
Thus, the apparent size of targets increases with outward accommodation. However, with the 
use of a virtual image accommodation shifts inward toward the dark focus causing a decrease in 
retinal image size thus, apparent size or accommodation micropsia. The Zoom-lens hypothesis 
implies that changes in retinal image size are not necessarily related to focal distance. This raises 
the question of how size constancy is mediated with virtual image displays. 

Other researchers (Marsh & Temme, 1990; Smith, Meehan,& Day, 1992) argue that the 
reduction in apparent size of objects can not be attributed to a change in retinal image size 
caused by accommodation. Equations, derived from studies utilizing a schematic eye, show that 
retinal image size decreases about 2% with 10D accommodation, or 0.3% at 1.5D, the 
approximate dark focus of accommodation. These changes arc not large enough to explain the 
minification of objects (Roscoe, 1984) viewed through a virtual image display. In normal 
viewing circumstances, accommodation and convergence are linked to present a single fused and 
focused image on the retina. Convergence stimulates accommodation. Accommodation also 
stimulates convergence. This is accommodative convergence. Smith et al (1992) suggest that 
accommodation micropsia is a result of accommodative convergence. An increase in 
convergence can decreased the perceived size of an object (Heinemann, Tulving, & Nachmias, 
1959) particularly when the vergence is induced by viewing through a prism. However, the 
increase in convergence necessary to induce the size change may also produce diplopia or double 
vision. It is plausible that accommodation micropsia can result due to an increase in 
accommodative convergence. 

Convergence is necessary to fuse the images in a stereo display. However, convergence 
and accommodation cues may be decoupled when viewing objects in the virtual display. It is 
assumed that accommodation is fixed for all objects regardless of the distance at which objects 
are presented through the helmet. For example, when two objects exist at two depths in the 
virtual display the observer will see double images of the unattended object because both objects 
are in focus simultaneously. The observer cannot converge on both objects simultaneously 
unless they are in close range. Thus, the feedback loops between accommodation and 
convergence are interrupted with the stereo virtual image display. Fusional convergence does not 



influence accommodation, likewise accommodative convergence is not an influence because 
accommodation does not change. Numerous studies have shown that when accommodation is 
not possible, size judgments made with the convergence cue are underestimated. Consequently, 
distance judgments are also inaccurate. It is possible, that using stereo virtual displays may 
result in inaccurate size and distance judgments because accommodation is fixed and size 
judgments are based on the convergence cue. The display will not eliminate the accommodation 
micrposia problem. 

Size Constancy 

The presentation of pictorial depth cues in the display may facilitate the accuracy of 
distance and size judgments. In normal viewing conditions, as the distance between the eye and a 
target increases the size of the retinal image decreases. However, the perception of size does not 
change as retinal image size changes. This phenomenon is size constancy. Thus, changes in the 
perception of apparent distance should alter judgments in apparent size. Therefore, the observer’s 
ability to make veridical size judgments depends on the accuracy of the observer’s distance 
judgments. The law of size constancy implies that the observer makes size judgments based on 
the size and distance of a physical object, or the distal stimulus, rather than retinal image size, or 
the proximal stimulus. When adequate distance cues such as, linear perspective, texture 
gradient, optical flow patterns, convergence, accommodation, binocular disparity and head 
movement parallax are present and in agreement the observer relies on the distal stimulus to 
make size judgments. However, when these cues are diminished or unavailable, the observer 
relies on the proximal stimulus or visual angle to judge size. 

For example, a classic experiment by Holway and Boring (1941) showed size judgments 
were based on the distal stimulus when distance cues were available. That is, as a target stimuli 
changed in distance, the observers changed the physical size of a stimulus to match the size of 
the target stimulus. In a dark room with reduced distance cues, observers made size judgments 
based on visual angle or retinal image size. That is, observers did not perceive a change in size of 
the stimulus because it always subtended the same visual angle. 

Previous research (Meehan & Triggs, 1992) has also shown visual scenes with plentiful 
depth cues result in more accurate judgments of size. Meehan and Triggs (1992) argue that 
apparent size and apparent distance are determined separately and independently by the visual 
system. They showed that estimated size was close to actual size when multiple dies to depth 
were available. However, perceived distance was not influenced by the absence of depth cues in 
a scene. They concluded distance cues facilitate size judgments more than distance cues facilitate 
distance judgments. Likewise, size is an important cue for distance. Therefore, including depth 
cues such as, linear perspective, texture gradient, and interposition within the virtual image 
display may result in more veridical size judgments, thus more accurate distance judgments. 
Modeling the physical world’s dimensions in a display and providing appropriate depth cues 



within the display should produce more accurate size and distance judgments. This is a unique 
advantage of stereo virtual images. 

Meehan and Triggs (1992) concluded that different cues or sets of cues may influence 
differentially perceptions of size and distance. For example, Thouless (1968) suggests that 
binocular disparity influences distance judgments more than it influences size judgments, and 
monocular viewing influences size judgments more than distance judgments. Furthermore, the 
importance of different cues will vary based on viewing conditions. Iavecchia et al. (1988) have 
shown that the background texture and acuity demand of a task can affect size and distance 
judgments. Therefore, the conditions which facilitate or hinder size and distance judgments need 
to be examined and understood. Display designers of virtual image displays need to consider the 
visual task requirements, the scene or terrain content, ambient lighting, viewing distance, color, 
the mode of viewing; monocular, biocular, or binocular modes, and other factors that define 
viewing conditions. If stereo virtual image displays eliminate the accommodation micropsia 
problem or mediate size constancy, then they have a performance advantage over traditional 
biocular and two-dimensional virtual image displays. 

This research was done to examine the effects that various viewing conditions have on 
distance judgments and size judgments with the hope of determining the performance advantages 
of stereo virtual image displays. These studies attempted to answer the following questions: (1) 
Will accommodation micropsia occur when viewing an object through a three-dimensional 
stereo helmet mounted display? (2) Does presentation of distance cues with and without size 
cues in the virtual image display influence the accuracy of size and distance judgments? (3) 
Which distance cues or combinations of cues are more important when judging distance and 
size? (4) Does the type of background of the physical world interact to influence the accuracy of 
size and distance judgments? (5) Will size constancy be mediated similarly inside the range of 
convergence, less than 20 feet, and outside the range of convergence? 

Participants were required to move virtual image objects to match the depth and size of 
physical objects. Participants will perform the judgments viewing a biocular display, then 
viewing a binocular display. Half the participants will have a size cue when making distance 
judgments; the visual angle of the virtual object will change according to the distance moved. 
The other participants will not have a size cue; the virtual image will subtend the same visual 
angle as it is moved in depth. Participants will make judgments in situations which vary the 
number of distance cues from one cue, accommodation, to a full cue condition including 
accommodation, retinal disparity, convergence, linear and texture perspectives, and occlusion. 
However, a target detection study (Fulop & Williams, 1993) suggested that accurate perception 
of distance in the helmet is not possible with accommodation, retinal disparity, and convergence 
alone. Pictorial cues are needed. Distance and size judgments will be made against two types of 
backgrounds, textured and untextured. 

Hyp otheses 
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Accommodation will be fixed near or at the dark focus when accommodation is the only 
cue present in the display. Participants viewing a physical textured background will be 
accommodated closer to the dark focus than participants not viewing a physical textured 
background. Thus, participants viewing a textured background should underestimate size and 
overestimate distance compared to non-textured background participants. Furthermore, 
judgments of apparent size should not be affected by the distance of the target. There should be 
no difference in size judgments between those participants who receive a distance cue and those 
who do not. Distance judgments will be based on the change in visual angle of the object. 
Therefore, distance judgments will be impossible to make for those participants who do not 
receive a size cue. 

Because accommodation and convergence are decoupled with the display, size judgments 
should be underestimated for all participants. Therefore, distance judgments should be 
inaccurate. However, the binocular disparity cue may interact with the convergence cue to 
influence distance judgments so that the judgments are more accurate. 

Adding linear perspective and texture gradient to the display should result in more 
accurate size and distance judgments for all participants. 

METHOD 

Participants 

Thirty-two participants between the ages of 20 years and 40 years will volunteer to 
participate in the study. Participants will be tested for visual acuity with an orthorater and tested 
for stereoacuity with a random dot stereogram test. Participants’ interpupillary distances (IPD) 
will be measured. All participants will need IPD’s greater than 56 mm. 

Procedure 

After the helmet is properly fit on participants, the instructions will be read to them. 

Then, they will make distance and size judgments in both biocular and binocular display 
conditions. The order for distance-size judgments and biocular-binocular displays will be 
counterbalanced. When making judgments cues will be presented in an additive or subtractive 
manner. This will also be counterbalanced. That is, in the additive condition participants will 
begin with the accommodation cue, make the distance judgments then will be presented with 
accommodation and retinal disparity cues, and eventually followed by the full cue conditions of 
accommodation, retinal disparity, convergence, linear perspective, texture gradient, and 
interposition. In the subtractive condition, participants will begin with the full cue condition, 
then cues will be eliminated one by one until accommodation will be the last cue. 

Participants will make 4 distance judgments with each set of cues, then the next set of 
cues will be presented. Each participant will make 24 distance judgments in the binocular display 
conditions and 20 distance judgments in the biocular display condition. Participants will make 4 
size judgments with each set of cues. Again, cues will be presented in an additive or subtractive 
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manner. Each participant will make 24 size judgments in the binocular display condition and 20 
size judgments in the biocular display conditions. A participant will receive the same order, 
additive or subtractive, of cue presentation for both the distance and size judgments. 

Inde p endent Variables and Ex p erimental Design 

The six independent variables in the study include size cue presentation, depth cue 
presentation, type of physical background, distance of physical stimuli from observer, the size of 
physical stimuli, and the mode of display presentation. Size cue presentation will be 
manipulated as a between subjects variable when participants make distance judgments. Half the 
participants will receive a size cue when making distance judgments. The size of the virtual 
object will be adjusted as the object moves through space, therefore size constancy will be 
maintained. The remaining participants will not receive a size cue when making distance 
judgments. The visual angle subtended of the virtual stimulus will remain constant, therefore size 
constancy will not be maintained. 

When making distance judgments, the physical stimuli will be placed at four distances 

feet and feet, outside of the range of convergence, and feet and 

feet, inside the range of convergence. The farthest physical stimulus will be placed at the same 
depth as the physical background. 

Depth cue will be manipulated as a between subjects variable when participants make 

size judgments. The physical and virtual stimuli will be placed at the same depth, , for 

half the participants. The remaining participants will be presented with the physical and virtual 

stimuli placed feet apart with the virtual stimulus displayed at feet and the 

physical stimulus placed at feet 

When making size judgments, the 4 visual angle sizes of the physical stimuli will be , 

Mode of presentation will be manipulated as a within subjects variable with 2 levels. 
Participants will first make their distance and size judgments viewing a biocular display. 

Identical images of the virtual image will be presented to each eye. Then, participants will repeat 
the judgments viewing a binocular display, in which two slightly different images will be 
presented to the observer. The observer will need to fuse the images into a single image. 

Distance Cues 

The six depth cues included in the study include: accommodation(A);retinal 
disparity(RD) and A; convergence(C), RD and A; linear perspective(LP), C, RD, and A; texture 
gradient (TG). LP, C, RD, and A; occlusion(O), TG, LP, C, RD, and A. The binocular mode of 
presentation includes all of the above cues. The biocular mode of presentation eliminates the 
convergence cue in all cases. It is not necessary to manipulate independently accommodation, 
retinal disparity, and convergence. A previous study (Fulop & Williams, 1993) has shown that 
distance is not accurately perceived with these cues alone. 
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Linear perspective will be presented as a stereo display in the binocular condition and as 
a three-dimensional non stereo display in the biocular condition. Texture gradient will be 
presented as a stereo display in the binocular condition and as a three-dimensional display in the 
biocular condition. Occlusion or interposition will be presented as a stereo display in the 
binocular condition and as a three-dimensional non stereo display in the biocular condition. 
Performance Measures 

Performance measures for distance judgments will be the distance traversed for the 
virtual image and the difference in distance between the physical stimulus and the virtual 
stimulus. Distance should be measured in feet, pixels displaced, and change in retinal disparity. 
Several measures are needed because it is unknown whether the stereo equations generating the 
stereo images are correct. Electronic measures of distance in the helmet have to be related to 
physical measures of distance. 

Performance measures for size judgments will be the size change in the virtual image and 
the difference in size between the physical stimulus and the virtual stimulus. Size should be 
measured in inches and pixels displaced. 
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NOTES 

The distance study will need to be done outside. It should be completed as soon as the 
helmet is operational and weather permits. This research should tell researcher whether the 
stereo equations used to generate the images are correct. It will also tell the researcher the 
available volume of the display. If the volume is less than 1000 feet, the helmet will not be 
sufficient for air combat. The displays are programmed. They are available from Steve Williams, 
NASA-Langely Research Center, Hampton VA. 
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FUTURE RESEARCH 

Interposition problem 

Interposition cues between the virtual objects and physical objects are not predictable. 
Virtual objects do not behave like physical objects. Virtual objects can pass through physical 
objects or be superimposed on objects. This problem needs to be explored to determine the 
factors that influence perceptions of placement or location of virtual objects, and to determine 
types of errors made from misperceptions. Solutions to this problem should be evaluated. 

Field-of-View Studies 

Research determining the minimum acceptable field-of-view for helmet displays for 
various tasks needs to be determined. The tasks should include pursuit tracking, compensatory 
tracking, target detection, and spatial awareness tasks. It is assumed that the acceptable 
field-of-view will varying depending upon the stereo viewing volume of the display. For 
example, if virtual objects are going to be presented between 1000 - 3000 feet the field-of-view 
may be different than when the volume is 50 - 500 feet. However, the helmet has a maximum 
field-of-view of 60 degrees, therefore not all the research can be done with the helmet. Some of 
the research can be done in a University laboratory. 
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Long term exposure study 

The long term effects of viewing the stereo helmet mounted display on stereoacuity or 
depth perception is unknown. It is foreseeable that pilots will be wearing these helmets on a 
daily basis for hours at a time. The long term effects of decoupling convergence and 
accommodation should be explored. 


Conformal Displays 

At NASA-Langley we have defined a conformal display to be a representation of a 
concept presented in a location either relative to the earth, air, pilot’s aircraft, or opponent’s 
aircraft. Conformal displays make an abstract concept concrete by providing a visual 
representation of the concept to the pilot. Therefore, the pilot does not have to mentally map a 
display into a 3-D space or infer system states from several alphanumeric values. The pilot can 
"see" this information in a conformal display, thus cognitive workload should be reduced. 
Disadvantage of a conformal display is that the pilot must look in the correct location for 
information thus, information is not always in the center of the field of view. Research needs to 
be done in this area to answer the following questions. 

A) Which type of display is better for navigating a safe path through air space a 2-D 
bird’s eye view or a 3-D conformal display? 

B) Which type of display is better for troubleshooting and monitoring aircraft systems, 
such as fuel consumption, traditional cockpit gauges or gauges conformal to the aircraft? For 
example, the pilot would look over the shoulder at fuel tank to "see" quantity of fuel in tank. 

C) Does a 3-D conformal display prevent pilots from becoming disoriented or help them 
recover from unusual attitudes more quickly because the helmet can account for head position 
and adjust display accordingly? 


Attention Study 

The target detection study (Fulop & Williams, 1993) suggested that it may be difficult for 
pilots to switch attention between the physical world and the helmet world. Research addressing 
this issue needs to be completed. The following questions needs to be addressed. 

(1) Is it difficult for pilots to switch attention from the virtual world to the physical world? OR, 
from the helmet world to head down displays? 

(2) As the virtual world begins to look more like the physical world, will switching attention 
become more difficult or less difficult? Or, will the task change. That is, everything may look 
physical, so pilots would have to only switch attention form one object in the "real world" to 
another object in the real world. 

All research should result in information that can be used to write display guidelines for 
display designers. 


